<p><strong>This is a Reference Page:</strong> Reference pages are collections of short bits
of information that are useful to look up but not long or interesting enough to
justify having their own dedicated post.</p>
<p>This reference page lists some of the errors that I've personally encountered,
and how I fixed them. I might also include errors I never fixed in case anyone
wants to talk about it.</p>
<hr>
<h3>Table of Contents</h3>
<ul>
<li><a href="#docker-login-fails">Docker Login Fails</a></li>
<li><a href="#runaway-memory-usage-in-ceph-osd-process">Runaway memory usage in Ceph OSD process</a></li>
<li><a href="#chrome-blocks-https127001-with-neterr_cert_revoked">Chrome blocks https://127.0.0.1 with NET::ERR_CERT_REVOKED</a></li>
<li><a href="#failed-command-aws-ecr-get-login">Failed command: aws ecr get-login</a></li>
<li><a href="#pure-storage-iscsi-with-kolla-openstack---fdisk-freezes">Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</a></li>
<li><a href="#weird-chars-when-pasting-on-mac-os">Weird chars when pasting on Mac OS</a></li>
</ul>
<h1>Docker Login Fails</h1>
<pre><code>Error saving credentials: error storing credentials - err: exit status 1, out: `Cannot autolaunch D-Bus without X11 $DISPLAY`
</code></pre>
<p>I had this happen after running both Ceph-Ansible then Kolla-Ansible on an Ubuntu server, so it could be either of those that set up the conditions for this to happen. <a href="https://github.com/docker/cli/issues/1136">this solution</a> worked for me:</p>
<pre><code class="language-bash">apt-get install -y gnupg2 pass
</code></pre>
<hr>
<h1>Runaway memory usage in Ceph OSD process</h1>
<p>It looked like a memory leak. Each OSD process was using around 30GB of RAM.</p>
<p>It turns out that Ceph-Ansible has the following logic when generating its
<code>ceph.conf</code> file:</p>
<pre><code># this math
roles/ceph-config/templates/ceph.conf.j2:{% set _osd_memory_target = (ansible_memtotal_mb * 1048576 * hci_safety_factor / _num_osds) | int %}
# sets osd memory target
roles/ceph-config/templates/ceph.conf.j2:osd memory target = {{ _osd_memory_target | default(osd_memory_target) }}
</code></pre>
<p>In my case, this set the value WAY too high. In one of my deployments, it was
set to <code>37853425827</code>, which is like 35GB (PER OSD SERVICE!).</p>
<p>The 4GB default is much more sane, <code>4294967296</code>.</p>
<pre><code># ceph.conf
[osd]
osd memory target = 4294967296
</code></pre>
<hr>
<h1>Chrome blocks <a href="https://127.0.0.1">https://127.0.0.1</a> with NET::ERR_CERT_REVOKED</h1>
<p>This comes up when I reverse tunnel to a remove web server, forwarding it to my workstation. Chrome has decided to not let you do that. They give this errror:</p>
<pre><code class="language-text">Your connection is not private
Attackers might be trying to steal your information from 127.0.0.1 (for example, passwords, messages, or credit cards). Learn more
NET::ERR_CERT_REVOKED
</code></pre>
<p>To bypass that, open <code>chrome://flags/#allow-insecure-localhost</code> and set allow-insecure-localhost to enabled. Click relaunch. This will nuke your current browser…</p>
<p>Once it relaunches you'll be able to open the site.</p>
<hr>
<h1>Failed command: aws ecr get-login</h1>
<p>This error was caused by a bad NTP config. Renew the NTP lease to fix it.</p>
<pre><code>An error occurred (InvalidSignatureException) when calling the GetAuthorizationToken operation: Signature expired: 20190911T174538Z is now earlier than 20190912T004835Z (20190912T010335Z - 15 min.)
</code></pre>
<hr>
<h1>Pure Storage iSCSI with Kolla OpenStack - fdisk freezes</h1>
<p>This was a weird one. When using the Kolla containers project to make a
Cinder-volume container that has the <code>purecinder</code> plugin, I was able to get
OpenStack to create the volumes in Pure but it couldn't mount them to the hosts
for things like creating a volume with the <code>--image</code> argument. That made the
volumes I could create basically useless.</p>
<p>These errors were in the logs:</p>
<pre><code class="language-text"># Lots of these in cinder's log
Trying to connect to iSCSI portal

# also from cinder's log, also recurring:
iscsiadm stderr output when getting sessions: iscsiadm: No active sessions

# in DMESG
FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
</code></pre>
<p>The weirdest part was that the <code>fdisk -l</code> command would totally lock up and
the process running it couldn't be killed even with <code>kill -9</code>. The iSCSI volume
never got properly mounted but Pure support said they could see some data
coming into it.</p>
<p><strong>Root cause</strong>: MTU mismatch. Of course. I had set MTU 9000 on the iSCSI ports,
my host ports, and the switchports heading to the Pure Storage iSCSI
interfaces. I had forgotten to set it on the switch heading to the Dell servers
being used for OpenStack.</p>
<hr>
<h1>Weird chars when pasting on Mac OS</h1>
<p>When you paste sometimes you get <code>^[[200~</code> at the start and <code>^[[201~</code> at the
end of the pasted content. Or maybe <code>0~</code> and <code>1~</code>.</p>
<p>To resolve, run:</p>
<pre><code class="language-bash">printf "\e[?2004l"^C
</code></pre>


Kyle Pericak

"It works in my environment"

Categories

Tags